OpenAI

The OpenAI class can be used to access OpenAI models like GPT-4, GPT-4 Turbo, and GPT-3.5 Turbo.

OpenAI compatible servers / APIs

OpenAI compatible servers

• KoboldCpp
• text-gen-webui
• FastChat
• LocalAI
• llama-cpp-python
• TensorRT-LLM

OpenAI compatible APIs

• Anyscale Endpoints
• Anyscale Private Endpoints

If you are using an OpenAI compatible server / API, you can change the apiBase like this:

~/.continue/config.json

{
  "models": [
    {
      "title": "OpenAI-compatible server / API",
      "provider": "openai",
      "model": "MODEL_NAME",
      "apiKey": "EMPTY",
      "apiBase": "http://localhost:8000/v1"
    }
  ]
}

To force usage of chat/completions instead of completions endpoint you can set

"useLegacyCompletionsEndpoint": false

View the source